{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "// %install-swiftpm-flags -c release\n",
    "// %install '.package(url: \"https://github.com/JacopoMangiavacchi/SwiftCoreMLTools.git\", from: \"0.0.5\")' SwiftCoreMLTools\n",
    "// %install '.package(url: \"https://github.com/dduan/Just.git\", from: \"0.8.0\")' Just"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TO4FT9bUohHx"
   },
   "outputs": [],
   "source": [
    "import Foundation\n",
    "import TensorFlow\n",
    "// import SwiftCoreMLTools\n",
    "// import Just"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Download\n",
    "\n",
    "Boston house prices dataset\n",
    "---------------------------\n",
    "\n",
    "**Data Set Characteristics:**  \n",
    "\n",
    "    :Number of Instances: 506 \n",
    "\n",
    "    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\n",
    "\n",
    "    :Attribute Information (in order):\n",
    "        - CRIM     per capita crime rate by town\n",
    "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
    "        - INDUS    proportion of non-retail business acres per town\n",
    "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
    "        - NOX      nitric oxides concentration (parts per 10 million)\n",
    "        - RM       average number of rooms per dwelling\n",
    "        - AGE      proportion of owner-occupied units built prior to 1940\n",
    "        - DIS      weighted distances to five Boston employment centres\n",
    "        - RAD      index of accessibility to radial highways\n",
    "        - TAX      full-value property-tax rate per ten thousand dollars\n",
    "        - PTRATIO  pupil-teacher ratio by town\n",
    "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
    "        - LSTAT    % lower status of the population\n",
    "        - MEDV     Median value of owner-occupied homes in a thousand dollar\n",
    "\n",
    "    :Missing Attribute Values: None\n",
    "\n",
    "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
    "\n",
    "This is a copy of UCI ML housing dataset.\n",
    "https://archive.ics.uci.edu/ml/machine-learning-databases/housing/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "// if let cts = Just.get(URL(string: \"https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.data\")!).content {\n",
    "//     try! cts.write(to: URL(fileURLWithPath:\"../data/housing.csv\"))\n",
    "// }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Ingestion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "let data = try String(contentsOfFile:\"../data/housing.csv\", encoding: String.Encoding.utf8)\n",
    "let dataRecords: [[Float]] = data.split(separator: \"\\n\").map{ String($0).split(separator: \" \").compactMap{ Float(String($0)) } }\n",
    "\n",
    "let numRecords = dataRecords.count\n",
    "let numColumns = dataRecords[0].count\n",
    "\n",
    "var index = Set<Int>()\n",
    "\n",
    "while index.count < numRecords {\n",
    "    index.insert(Int.random(in: 0..<numRecords))\n",
    "}\n",
    "\n",
    "let randomDataRecords = index.map{ dataRecords[$0] }\n",
    "\n",
    "let dataFeatures = randomDataRecords.map{ Array($0[0..<numColumns-1]) }\n",
    "let dataLabels = randomDataRecords.map{ Array($0[(numColumns-1)...]) }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split Numerical Categorical Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "let categoricalColumns = [3, 8]\n",
    "let numericalColumns = [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 12]\n",
    "let numCategoricalFeatures = categoricalColumns.count\n",
    "let numNumericalFeatures = numericalColumns.count\n",
    "let numLabels = 1\n",
    "\n",
    "assert(numColumns == numCategoricalFeatures + numNumericalFeatures + 1)\n",
    "\n",
    "// Get Categorical Features\n",
    "let allCategoriesValues = dataFeatures.map{ row in categoricalColumns.map{ Int32(row[$0]) } }\n",
    "                                .reduce(into: Array(repeating: [Int32](), count: 2)){ total, value in\n",
    "                                    total[0].append(value[0])\n",
    "                                    total[1].append(value[1]) }\n",
    "                                .map{ Set($0).sorted() }\n",
    "\n",
    "let categoricalFeatures = dataFeatures.map{ row in categoricalColumns.map{ Int32(row[$0]) } }\n",
    "\n",
    "// Get Numerical Features\n",
    "let numericalFeatures = dataFeatures.map{ row in numericalColumns.map{ row[$0] } }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorize Categorical Features with Ordinal values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "var categoricalValues = Array(repeating: Set<Int32>(), count: 2)\n",
    "\n",
    "for record in categoricalFeatures {\n",
    "    categoricalValues[0].insert(record[0])\n",
    "    categoricalValues[1].insert(record[1])\n",
    "}\n",
    "\n",
    "let sortedCategoricalValues = [categoricalValues[0].sorted(), categoricalValues[1].sorted()]\n",
    "\n",
    "let ordinalCategoricalFeatures = categoricalFeatures.map{ [Int32(sortedCategoricalValues[0].firstIndex(of:$0[0])!), \n",
    "                                                           Int32(sortedCategoricalValues[1].firstIndex(of:$0[1])!)] }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1] [1, 2, 3, 4, 5, 6, 7, 8, 24]\r\n",
      "24.0\r\n",
      "[9.92485, 0.0, 18.1, 0.74, 6.251, 96.6, 2.198, 666.0, 20.2, 388.52, 16.44]\r\n",
      "[9.92485, 0.0, 18.1, 0.0, 0.74, 6.251, 96.6, 2.198, 24.0, 666.0, 20.2, 388.52, 16.44]\r\n",
      "[0, 24]\r\n",
      "[0, 8]\r\n"
     ]
    }
   ],
   "source": [
    "print(allCategoriesValues[0], allCategoriesValues[1])\n",
    "print(dataFeatures[9][8])\n",
    "print(numericalFeatures[9])\n",
    "print(dataFeatures[9])\n",
    "print(categoricalFeatures[9])\n",
    "print(ordinalCategoricalFeatures[9])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split Train and Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "let trainPercentage:Float = 0.8\n",
    "let numTrainRecords = Int(ceil(Float(numRecords) * trainPercentage))\n",
    "let numTestRecords = numRecords - numTrainRecords\n",
    "\n",
    "func matrixTranspose<T>(_ matrix: [[T]]) -> [[T]] {\n",
    "    if matrix.isEmpty {return matrix}\n",
    "    var result = [[T]]()\n",
    "    for index in 0..<matrix.first!.count {\n",
    "        result.append(matrix.map{$0[index]})\n",
    "    }\n",
    "    return result\n",
    "}\n",
    "\n",
    "let xCategoricalAllTrain = matrixTranspose(Array(ordinalCategoricalFeatures[0..<numTrainRecords]))\n",
    "let xCategoricalAllTest = matrixTranspose(Array(ordinalCategoricalFeatures[numTrainRecords...]))\n",
    "let xNumericalAllTrain = Array(Array(numericalFeatures[0..<numTrainRecords]).joined())\n",
    "let xNumericalAllTest = Array(Array(numericalFeatures[numTrainRecords...]).joined())\n",
    "let yAllTrain = Array(Array(dataLabels[0..<numTrainRecords]).joined())\n",
    "let yAllTest = Array(Array(dataLabels[numTrainRecords...]).joined())\n",
    "\n",
    "let XCategoricalTrain = xCategoricalAllTrain.enumerated().map{ (offset, element) in \n",
    "    Tensor<Int32>(element).reshaped(to: TensorShape([numTrainRecords, 1]))\n",
    "}\n",
    "let XCategoricalTest = xCategoricalAllTest.enumerated().map{ (offset, element) in \n",
    "    Tensor<Int32>(element).reshaped(to: TensorShape([numTestRecords, 1]))\n",
    "}\n",
    "\n",
    "let XNumericalTrainDeNorm = Tensor<Float>(xNumericalAllTrain).reshaped(to: TensorShape([numTrainRecords, numNumericalFeatures]))\n",
    "let XNumericalTestDeNorm = Tensor<Float>(xNumericalAllTest).reshaped(to: TensorShape([numTestRecords, numNumericalFeatures]))\n",
    "let YTrain = Tensor<Float>(yAllTrain).reshaped(to: TensorShape([numTrainRecords, numLabels]))\n",
    "let YTest = Tensor<Float>(yAllTest).reshaped(to: TensorShape([numTestRecords, numLabels]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalize Numerical Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  3.837692,  10.720987,   11.27075, 0.55511284,  6.2830257,    68.3573,   3.760012,    407.558,\n",
      "   18.485954,   359.1895, 12.7277775]] [[  9.297075,  22.210146,  6.8542833, 0.11656232,  0.7115993,  28.457472,  2.0794964,  169.48473,\n",
      "    2.119372,  86.309074,  7.3216815]]\n"
     ]
    }
   ],
   "source": [
    "let mean = XNumericalTrainDeNorm.mean(alongAxes: 0)\n",
    "let std = XNumericalTrainDeNorm.standardDeviation(alongAxes: 0)\n",
    "\n",
    "print(mean, std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "let XNumericalTrain = (XNumericalTrainDeNorm - mean)/std\n",
    "let XNumericalTest = (XNumericalTestDeNorm - mean)/std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training shapes [405, 11] [405, 1] [405, 1] [405, 1]\r\n",
      "Testing shapes  [101, 11] [101, 1] [101, 1] [101, 1]\r\n"
     ]
    }
   ],
   "source": [
    "print(\"Training shapes \\(XNumericalTrain.shape) \\(XCategoricalTrain[0].shape) \\(XCategoricalTrain[1].shape) \\(YTrain.shape)\")\n",
    "print(\"Testing shapes  \\(XNumericalTest.shape) \\(XCategoricalTest[0].shape) \\(XCategoricalTest[1].shape) \\(YTest.shape)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HDJZCCgqohIC"
   },
   "outputs": [],
   "source": [
    "struct MultiInputs<N: Differentiable, C>: Differentiable {\n",
    "  var numerical: N\n",
    "  \n",
    "  @noDerivative\n",
    "  var categorical: C\n",
    "\n",
    "  @differentiable\n",
    "  init(numerical: N, categorical: C) {\n",
    "    self.numerical = numerical\n",
    "    self.categorical = categorical\n",
    "  }\n",
    "}\n",
    "\n",
    "struct RegressionModel: Module {\n",
    "    var numericalLayer = Dense<Float>(inputSize: 11, outputSize: 32, activation: relu)\n",
    "    var embedding1 = Embedding<Float>(vocabularySize: 2, embeddingSize: 2)\n",
    "    var embedding2 = Embedding<Float>(vocabularySize: 9, embeddingSize: 5)\n",
    "    var embeddingLayer = Dense<Float>(inputSize: (2 + 5), outputSize: 32, activation: relu)\n",
    "    var allInputConcatLayer = Dense<Float>(inputSize: (32 + 32), outputSize: 128, activation: relu)\n",
    "    var hiddenLayer = Dense<Float>(inputSize: 128, outputSize: 32, activation: relu)\n",
    "    var outputLayer = Dense<Float>(inputSize: 32, outputSize: 1)\n",
    "    \n",
    "    @differentiable\n",
    "    func callAsFunction(_ input: MultiInputs<[Tensor<Float>], [Tensor<Int32>]>) -> Tensor<Float> {\n",
    "        let numericalInput = numericalLayer(input.numerical[0])\n",
    "        let embeddingOutput1 = embedding1(input.categorical[0])\n",
    "        let embeddingOutput1Reshaped = embeddingOutput1.reshaped(to: \n",
    "            TensorShape([embeddingOutput1.shape[0], embeddingOutput1.shape[1] * embeddingOutput1.shape[2]]))\n",
    "        let embeddingOutput2 = embedding2(input.categorical[1])\n",
    "        let embeddingOutput2Reshaped = embeddingOutput2.reshaped(to: \n",
    "            TensorShape([embeddingOutput2.shape[0], embeddingOutput2.shape[1] * embeddingOutput2.shape[2]]))\n",
    "        let embeddingConcat = Tensor<Float>(concatenating: [embeddingOutput1Reshaped, embeddingOutput2Reshaped], alongAxis: 1)\n",
    "        let embeddingInput = embeddingLayer(embeddingConcat)\n",
    "        let allConcat = Tensor<Float>(concatenating: [numericalInput, embeddingInput], alongAxis: 1)\n",
    "        return allConcat.sequenced(through: allInputConcatLayer, hiddenLayer, outputLayer)\n",
    "    }\n",
    "}\n",
    "\n",
    "var model = RegressionModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JK0Vj7bSohIF"
   },
   "outputs": [],
   "source": [
    "let optimizer = RMSProp(for: model, learningRate: 0.001)\n",
    "Context.local.learningPhase = .training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gY8C7yHJohIH"
   },
   "outputs": [],
   "source": [
    "let epochCount = 500\n",
    "let batchSize = 32\n",
    "let numberOfBatch = Int(ceil(Double(numTrainRecords) / Double(batchSize)))\n",
    "let shuffle = true\n",
    "\n",
    "func mae(predictions: Tensor<Float>, truths: Tensor<Float>) -> Float {\n",
    "    return abs(Tensor<Float>(predictions - truths)).mean().scalarized()\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 203
    },
    "colab_type": "code",
    "id": "L9bU9HsdohIK",
    "outputId": "692b81c5-3286-4e7c-9246-eb56e6a3eaee"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: MSE: 491.70114, MAE: 19.796753\n",
      "Epoch 2: MSE: 319.2449, MAE: 15.063409\n",
      "Epoch 3: MSE: 175.89833, MAE: 10.81574\n",
      "Epoch 4: MSE: 87.37532, MAE: 7.176715\n",
      "Epoch 5: MSE: 47.04053, MAE: 4.900528\n",
      "Epoch 6: MSE: 33.370583, MAE: 4.0233173\n",
      "Epoch 7: MSE: 28.9492, MAE: 3.6320593\n",
      "Epoch 8: MSE: 27.033323, MAE: 3.4715817\n",
      "Epoch 9: MSE: 25.723206, MAE: 3.3521864\n",
      "Epoch 10: MSE: 24.57535, MAE: 3.2877529\n",
      "Epoch 11: MSE: 23.5631, MAE: 3.1904118\n",
      "Epoch 12: MSE: 23.107466, MAE: 3.1184504\n",
      "Epoch 13: MSE: 22.688564, MAE: 3.0788329\n",
      "Epoch 14: MSE: 21.615004, MAE: 3.018838\n",
      "Epoch 15: MSE: 21.367666, MAE: 2.9685915\n",
      "Epoch 16: MSE: 20.618574, MAE: 2.93647\n",
      "Epoch 17: MSE: 20.335337, MAE: 2.905603\n",
      "Epoch 18: MSE: 19.758018, MAE: 2.8837278\n",
      "Epoch 19: MSE: 19.396109, MAE: 2.8424964\n",
      "Epoch 20: MSE: 19.329313, MAE: 2.8107526\n",
      "Epoch 21: MSE: 18.523045, MAE: 2.8074877\n",
      "Epoch 22: MSE: 18.752144, MAE: 2.774503\n",
      "Epoch 23: MSE: 18.14185, MAE: 2.7374887\n",
      "Epoch 24: MSE: 17.834278, MAE: 2.7134554\n",
      "Epoch 25: MSE: 17.616562, MAE: 2.6849096\n",
      "Epoch 26: MSE: 17.434155, MAE: 2.6745703\n",
      "Epoch 27: MSE: 16.98225, MAE: 2.634881\n",
      "Epoch 28: MSE: 16.27641, MAE: 2.612866\n",
      "Epoch 29: MSE: 16.175379, MAE: 2.563914\n",
      "Epoch 30: MSE: 16.049759, MAE: 2.5654554\n",
      "Epoch 31: MSE: 15.671017, MAE: 2.5340686\n",
      "Epoch 32: MSE: 15.574233, MAE: 2.485388\n",
      "Epoch 33: MSE: 14.809246, MAE: 2.4578645\n",
      "Epoch 34: MSE: 14.489234, MAE: 2.451132\n",
      "Epoch 35: MSE: 14.626547, MAE: 2.4061823\n",
      "Epoch 36: MSE: 14.413528, MAE: 2.3956735\n",
      "Epoch 37: MSE: 13.843618, MAE: 2.366201\n",
      "Epoch 38: MSE: 13.738795, MAE: 2.3358912\n",
      "Epoch 39: MSE: 13.437878, MAE: 2.3185525\n",
      "Epoch 40: MSE: 13.673804, MAE: 2.30022\n",
      "Epoch 41: MSE: 13.161516, MAE: 2.2851467\n",
      "Epoch 42: MSE: 13.0829525, MAE: 2.261393\n",
      "Epoch 43: MSE: 12.998506, MAE: 2.2356772\n",
      "Epoch 44: MSE: 12.570652, MAE: 2.200843\n",
      "Epoch 45: MSE: 12.195756, MAE: 2.1970606\n",
      "Epoch 46: MSE: 11.823539, MAE: 2.1927311\n",
      "Epoch 47: MSE: 12.173065, MAE: 2.1576152\n",
      "Epoch 48: MSE: 11.747427, MAE: 2.133065\n",
      "Epoch 49: MSE: 11.5424595, MAE: 2.1220174\n",
      "Epoch 50: MSE: 11.704752, MAE: 2.0938036\n",
      "Epoch 51: MSE: 11.017008, MAE: 2.0880046\n",
      "Epoch 52: MSE: 10.850745, MAE: 2.0569863\n",
      "Epoch 53: MSE: 11.118798, MAE: 2.068744\n",
      "Epoch 54: MSE: 10.633611, MAE: 2.0296755\n",
      "Epoch 55: MSE: 10.577178, MAE: 2.0203207\n",
      "Epoch 56: MSE: 10.1118765, MAE: 2.0047247\n",
      "Epoch 57: MSE: 10.333246, MAE: 1.9958524\n",
      "Epoch 58: MSE: 10.342813, MAE: 1.9929059\n",
      "Epoch 59: MSE: 9.927163, MAE: 1.9777044\n",
      "Epoch 60: MSE: 9.918917, MAE: 1.9526018\n",
      "Epoch 61: MSE: 9.881099, MAE: 1.9277039\n",
      "Epoch 62: MSE: 9.196792, MAE: 1.9086584\n",
      "Epoch 63: MSE: 9.686741, MAE: 1.9246379\n",
      "Epoch 64: MSE: 9.435043, MAE: 1.9175763\n",
      "Epoch 65: MSE: 9.1227455, MAE: 1.8926457\n",
      "Epoch 66: MSE: 8.886748, MAE: 1.8736333\n",
      "Epoch 67: MSE: 9.167651, MAE: 1.8782623\n",
      "Epoch 68: MSE: 8.581668, MAE: 1.85738\n",
      "Epoch 69: MSE: 9.03189, MAE: 1.8696867\n",
      "Epoch 70: MSE: 8.739738, MAE: 1.8478997\n",
      "Epoch 71: MSE: 8.457089, MAE: 1.8330743\n",
      "Epoch 72: MSE: 8.511599, MAE: 1.8098524\n",
      "Epoch 73: MSE: 8.733758, MAE: 1.8233262\n",
      "Epoch 74: MSE: 8.234095, MAE: 1.7967576\n",
      "Epoch 75: MSE: 8.20927, MAE: 1.8022214\n",
      "Epoch 76: MSE: 8.074785, MAE: 1.7692049\n",
      "Epoch 77: MSE: 8.135787, MAE: 1.7787443\n",
      "Epoch 78: MSE: 7.8623385, MAE: 1.7544097\n",
      "Epoch 79: MSE: 7.7792816, MAE: 1.7518044\n",
      "Epoch 80: MSE: 7.6793814, MAE: 1.7522779\n",
      "Epoch 81: MSE: 7.38541, MAE: 1.7493714\n",
      "Epoch 82: MSE: 7.494667, MAE: 1.6878664\n",
      "Epoch 83: MSE: 7.561515, MAE: 1.7240984\n",
      "Epoch 84: MSE: 7.2994165, MAE: 1.722111\n",
      "Epoch 85: MSE: 7.4276166, MAE: 1.7127687\n",
      "Epoch 86: MSE: 7.23227, MAE: 1.6933587\n",
      "Epoch 87: MSE: 6.929646, MAE: 1.6947051\n",
      "Epoch 88: MSE: 7.2423425, MAE: 1.6691573\n",
      "Epoch 89: MSE: 6.86749, MAE: 1.6467173\n",
      "Epoch 90: MSE: 6.678286, MAE: 1.6415594\n",
      "Epoch 91: MSE: 6.8115, MAE: 1.6676118\n",
      "Epoch 92: MSE: 6.8898163, MAE: 1.6328318\n",
      "Epoch 93: MSE: 6.907697, MAE: 1.6442707\n",
      "Epoch 94: MSE: 6.2771573, MAE: 1.6202419\n",
      "Epoch 95: MSE: 6.7152843, MAE: 1.6451797\n",
      "Epoch 96: MSE: 6.6730556, MAE: 1.627604\n",
      "Epoch 97: MSE: 6.5699263, MAE: 1.6023612\n",
      "Epoch 98: MSE: 6.321366, MAE: 1.5983107\n",
      "Epoch 99: MSE: 6.4265943, MAE: 1.5983747\n",
      "Epoch 100: MSE: 6.0133743, MAE: 1.5740814\n",
      "Epoch 101: MSE: 6.3344393, MAE: 1.5851668\n",
      "Epoch 102: MSE: 6.0692577, MAE: 1.5705909\n",
      "Epoch 103: MSE: 6.3350086, MAE: 1.5729337\n",
      "Epoch 104: MSE: 6.375443, MAE: 1.5764832\n",
      "Epoch 105: MSE: 6.1948323, MAE: 1.5557442\n",
      "Epoch 106: MSE: 5.965428, MAE: 1.5465317\n",
      "Epoch 107: MSE: 5.6690288, MAE: 1.5227847\n",
      "Epoch 108: MSE: 6.271599, MAE: 1.5491172\n",
      "Epoch 109: MSE: 5.3349466, MAE: 1.495647\n",
      "Epoch 110: MSE: 5.7394247, MAE: 1.5388211\n",
      "Epoch 111: MSE: 5.8705826, MAE: 1.5178615\n",
      "Epoch 112: MSE: 5.526408, MAE: 1.5019791\n",
      "Epoch 113: MSE: 5.886398, MAE: 1.5155358\n",
      "Epoch 114: MSE: 5.323057, MAE: 1.4923052\n",
      "Epoch 115: MSE: 5.7205997, MAE: 1.4819276\n",
      "Epoch 116: MSE: 5.7351427, MAE: 1.4907626\n",
      "Epoch 117: MSE: 5.3075027, MAE: 1.4933954\n",
      "Epoch 118: MSE: 5.8096623, MAE: 1.5254868\n",
      "Epoch 119: MSE: 5.118403, MAE: 1.4477376\n",
      "Epoch 120: MSE: 5.49202, MAE: 1.5084182\n",
      "Epoch 121: MSE: 5.315862, MAE: 1.4456098\n",
      "Epoch 122: MSE: 4.9039807, MAE: 1.4575081\n",
      "Epoch 123: MSE: 5.11015, MAE: 1.5033678\n",
      "Epoch 124: MSE: 5.2073774, MAE: 1.4607333\n",
      "Epoch 125: MSE: 5.284085, MAE: 1.4721026\n",
      "Epoch 126: MSE: 5.4714074, MAE: 1.4654666\n",
      "Epoch 127: MSE: 4.962883, MAE: 1.4467976\n",
      "Epoch 128: MSE: 5.223782, MAE: 1.426503\n",
      "Epoch 129: MSE: 5.2110677, MAE: 1.4138633\n",
      "Epoch 130: MSE: 5.141668, MAE: 1.4203527\n",
      "Epoch 131: MSE: 4.7735662, MAE: 1.412462\n",
      "Epoch 132: MSE: 5.2396116, MAE: 1.4225043\n",
      "Epoch 133: MSE: 4.858977, MAE: 1.4185741\n",
      "Epoch 134: MSE: 4.9297824, MAE: 1.3921884\n",
      "Epoch 135: MSE: 4.59091, MAE: 1.4044772\n",
      "Epoch 136: MSE: 4.611979, MAE: 1.3940029\n",
      "Epoch 137: MSE: 4.794719, MAE: 1.4534745\n",
      "Epoch 138: MSE: 4.7012854, MAE: 1.3580966\n",
      "Epoch 139: MSE: 4.8163137, MAE: 1.3709038\n",
      "Epoch 140: MSE: 4.934616, MAE: 1.4168645\n",
      "Epoch 141: MSE: 4.7213736, MAE: 1.3710726\n",
      "Epoch 142: MSE: 4.8690596, MAE: 1.3956361\n",
      "Epoch 143: MSE: 4.9921255, MAE: 1.3985751\n",
      "Epoch 144: MSE: 4.3280296, MAE: 1.3201267\n",
      "Epoch 145: MSE: 4.3664007, MAE: 1.3569382\n",
      "Epoch 146: MSE: 4.5596333, MAE: 1.3958855\n",
      "Epoch 147: MSE: 4.292107, MAE: 1.3585064\n",
      "Epoch 148: MSE: 4.989458, MAE: 1.3820821\n",
      "Epoch 149: MSE: 4.7013187, MAE: 1.3669207\n",
      "Epoch 150: MSE: 4.5594563, MAE: 1.3403748\n",
      "Epoch 151: MSE: 4.358501, MAE: 1.3364111\n",
      "Epoch 152: MSE: 4.0045958, MAE: 1.3333175\n",
      "Epoch 153: MSE: 4.185464, MAE: 1.4171963\n",
      "Epoch 154: MSE: 4.1938214, MAE: 1.3768686\n",
      "Epoch 155: MSE: 4.276401, MAE: 1.3787601\n",
      "Epoch 156: MSE: 4.6491184, MAE: 1.356191\n",
      "Epoch 157: MSE: 4.343175, MAE: 1.297675\n",
      "Epoch 158: MSE: 4.392608, MAE: 1.3110429\n",
      "Epoch 159: MSE: 3.827801, MAE: 1.2826914\n",
      "Epoch 160: MSE: 4.3505564, MAE: 1.3201735\n",
      "Epoch 161: MSE: 4.137308, MAE: 1.3366617\n",
      "Epoch 162: MSE: 4.3303714, MAE: 1.3049175\n",
      "Epoch 163: MSE: 4.0225606, MAE: 1.2873156\n",
      "Epoch 164: MSE: 4.3956304, MAE: 1.3268024\n",
      "Epoch 165: MSE: 4.181079, MAE: 1.2980502\n",
      "Epoch 166: MSE: 3.9563334, MAE: 1.2590404\n",
      "Epoch 167: MSE: 4.4432383, MAE: 1.3524461\n",
      "Epoch 168: MSE: 4.0533533, MAE: 1.2478033\n",
      "Epoch 169: MSE: 3.8588927, MAE: 1.2513807\n",
      "Epoch 170: MSE: 4.202695, MAE: 1.2803502\n",
      "Epoch 171: MSE: 4.2139187, MAE: 1.268395\n",
      "Epoch 172: MSE: 3.9720268, MAE: 1.2514167\n",
      "Epoch 173: MSE: 4.103769, MAE: 1.3053057\n",
      "Epoch 174: MSE: 3.8462398, MAE: 1.2710528\n",
      "Epoch 175: MSE: 3.6696055, MAE: 1.2994306\n",
      "Epoch 176: MSE: 3.7946088, MAE: 1.3310012\n",
      "Epoch 177: MSE: 4.163402, MAE: 1.2886778\n",
      "Epoch 178: MSE: 4.1607866, MAE: 1.2637621\n",
      "Epoch 179: MSE: 4.0098386, MAE: 1.2490745\n",
      "Epoch 180: MSE: 3.731909, MAE: 1.2791048\n",
      "Epoch 181: MSE: 3.713743, MAE: 1.2654502\n",
      "Epoch 182: MSE: 3.7874646, MAE: 1.2477746\n",
      "Epoch 183: MSE: 3.8252902, MAE: 1.2796847\n",
      "Epoch 184: MSE: 3.8877013, MAE: 1.2657577\n",
      "Epoch 185: MSE: 3.749631, MAE: 1.2640469\n",
      "Epoch 186: MSE: 3.7331252, MAE: 1.2371757\n",
      "Epoch 187: MSE: 3.9237885, MAE: 1.2753245\n",
      "Epoch 188: MSE: 3.638753, MAE: 1.2243342\n",
      "Epoch 189: MSE: 4.0061827, MAE: 1.2317796\n",
      "Epoch 190: MSE: 3.6541884, MAE: 1.211405\n",
      "Epoch 191: MSE: 3.801427, MAE: 1.2399701\n",
      "Epoch 192: MSE: 3.5442622, MAE: 1.2195266\n",
      "Epoch 193: MSE: 3.462991, MAE: 1.2358506\n",
      "Epoch 194: MSE: 3.7267342, MAE: 1.3113494\n",
      "Epoch 195: MSE: 3.441313, MAE: 1.1556647\n",
      "Epoch 196: MSE: 3.9550097, MAE: 1.2858783\n",
      "Epoch 197: MSE: 3.19326, MAE: 1.1704324\n",
      "Epoch 198: MSE: 4.1210833, MAE: 1.2551408\n",
      "Epoch 199: MSE: 3.220051, MAE: 1.15117\n",
      "Epoch 200: MSE: 3.8052142, MAE: 1.217767\n",
      "Epoch 201: MSE: 3.503234, MAE: 1.1940866\n",
      "Epoch 202: MSE: 3.3827188, MAE: 1.2299775\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 203: MSE: 3.7263958, MAE: 1.2213266\n",
      "Epoch 204: MSE: 3.5707762, MAE: 1.1892135\n",
      "Epoch 205: MSE: 3.454506, MAE: 1.2166088\n",
      "Epoch 206: MSE: 3.2851486, MAE: 1.1733152\n",
      "Epoch 207: MSE: 3.439362, MAE: 1.2310126\n",
      "Epoch 208: MSE: 3.265523, MAE: 1.268249\n",
      "Epoch 209: MSE: 3.4684894, MAE: 1.1564978\n",
      "Epoch 210: MSE: 3.268164, MAE: 1.1488271\n",
      "Epoch 211: MSE: 4.0976696, MAE: 1.288077\n",
      "Epoch 212: MSE: 2.9450924, MAE: 1.0932367\n",
      "Epoch 213: MSE: 3.1447053, MAE: 1.1818864\n",
      "Epoch 214: MSE: 3.64448, MAE: 1.254548\n",
      "Epoch 215: MSE: 3.4914217, MAE: 1.1432084\n",
      "Epoch 216: MSE: 3.2118149, MAE: 1.19139\n",
      "Epoch 217: MSE: 3.165588, MAE: 1.1283864\n",
      "Epoch 218: MSE: 3.1297069, MAE: 1.2087779\n",
      "Epoch 219: MSE: 3.702665, MAE: 1.2124261\n",
      "Epoch 220: MSE: 3.1566145, MAE: 1.1200536\n",
      "Epoch 221: MSE: 3.298924, MAE: 1.1428246\n",
      "Epoch 222: MSE: 3.3472965, MAE: 1.1950616\n",
      "Epoch 223: MSE: 2.900651, MAE: 1.1368037\n",
      "Epoch 224: MSE: 3.7760038, MAE: 1.207791\n",
      "Epoch 225: MSE: 2.934876, MAE: 1.0811876\n",
      "Epoch 226: MSE: 3.3302736, MAE: 1.1934303\n",
      "Epoch 227: MSE: 3.12623, MAE: 1.1362987\n",
      "Epoch 228: MSE: 2.911163, MAE: 1.147371\n",
      "Epoch 229: MSE: 2.8918874, MAE: 1.1154457\n",
      "Epoch 230: MSE: 3.110752, MAE: 1.2656335\n",
      "Epoch 231: MSE: 3.2505705, MAE: 1.1562495\n",
      "Epoch 232: MSE: 3.0279799, MAE: 1.0964894\n",
      "Epoch 233: MSE: 3.3071265, MAE: 1.1452718\n",
      "Epoch 234: MSE: 2.9387922, MAE: 1.0999599\n",
      "Epoch 235: MSE: 3.1997938, MAE: 1.1372339\n",
      "Epoch 236: MSE: 3.2671533, MAE: 1.10704\n",
      "Epoch 237: MSE: 3.0420065, MAE: 1.134621\n",
      "Epoch 238: MSE: 2.849848, MAE: 1.0933185\n",
      "Epoch 239: MSE: 3.304925, MAE: 1.1736089\n",
      "Epoch 240: MSE: 2.6959975, MAE: 1.0482287\n",
      "Epoch 241: MSE: 3.1711535, MAE: 1.2434291\n",
      "Epoch 242: MSE: 3.0988162, MAE: 1.1216441\n",
      "Epoch 243: MSE: 2.9757855, MAE: 1.0963278\n",
      "Epoch 244: MSE: 3.2253814, MAE: 1.212194\n",
      "Epoch 245: MSE: 2.840652, MAE: 1.0766019\n",
      "Epoch 246: MSE: 3.3002133, MAE: 1.1452628\n",
      "Epoch 247: MSE: 2.9972992, MAE: 1.0497886\n",
      "Epoch 248: MSE: 2.9432397, MAE: 1.1465689\n",
      "Epoch 249: MSE: 2.9559197, MAE: 1.1163588\n",
      "Epoch 250: MSE: 3.0353036, MAE: 1.1042253\n",
      "Epoch 251: MSE: 3.1688538, MAE: 1.1567216\n",
      "Epoch 252: MSE: 2.9837794, MAE: 1.0703187\n",
      "Epoch 253: MSE: 2.8754106, MAE: 1.0985532\n",
      "Epoch 254: MSE: 2.6553154, MAE: 1.0358663\n",
      "Epoch 255: MSE: 3.1656888, MAE: 1.2248052\n",
      "Epoch 256: MSE: 3.080577, MAE: 1.11545\n",
      "Epoch 257: MSE: 3.026192, MAE: 1.0733986\n",
      "Epoch 258: MSE: 2.5346985, MAE: 1.0248078\n",
      "Epoch 259: MSE: 3.2462454, MAE: 1.1978348\n",
      "Epoch 260: MSE: 2.739026, MAE: 1.1098437\n",
      "Epoch 261: MSE: 2.765427, MAE: 1.1032499\n",
      "Epoch 262: MSE: 3.037984, MAE: 1.1035287\n",
      "Epoch 263: MSE: 2.8984656, MAE: 1.1128473\n",
      "Epoch 264: MSE: 2.5384862, MAE: 1.0429581\n",
      "Epoch 265: MSE: 2.7050538, MAE: 1.1241524\n",
      "Epoch 266: MSE: 2.9792757, MAE: 1.1726744\n",
      "Epoch 267: MSE: 3.0315108, MAE: 1.098404\n",
      "Epoch 268: MSE: 2.7112257, MAE: 1.027779\n",
      "Epoch 269: MSE: 2.8938158, MAE: 1.119346\n",
      "Epoch 270: MSE: 2.7232134, MAE: 1.0385029\n",
      "Epoch 271: MSE: 2.9657533, MAE: 1.1852717\n",
      "Epoch 272: MSE: 2.547494, MAE: 1.047329\n",
      "Epoch 273: MSE: 2.9027352, MAE: 1.0931028\n",
      "Epoch 274: MSE: 2.7176056, MAE: 1.1032825\n",
      "Epoch 275: MSE: 3.0221934, MAE: 1.1761719\n",
      "Epoch 276: MSE: 2.4690783, MAE: 1.0263157\n",
      "Epoch 277: MSE: 3.0270734, MAE: 1.1578641\n",
      "Epoch 278: MSE: 2.669173, MAE: 1.0683764\n",
      "Epoch 279: MSE: 2.7640004, MAE: 1.0884099\n",
      "Epoch 280: MSE: 2.5568135, MAE: 1.0689774\n",
      "Epoch 281: MSE: 2.7898142, MAE: 1.1175485\n",
      "Epoch 282: MSE: 2.59487, MAE: 1.0686022\n",
      "Epoch 283: MSE: 2.1972218, MAE: 0.96138084\n",
      "Epoch 284: MSE: 3.0080156, MAE: 1.1140783\n",
      "Epoch 285: MSE: 2.8046165, MAE: 1.0614687\n",
      "Epoch 286: MSE: 2.3874598, MAE: 1.0099058\n",
      "Epoch 287: MSE: 2.8830707, MAE: 1.1281246\n",
      "Epoch 288: MSE: 2.4316478, MAE: 1.0074161\n",
      "Epoch 289: MSE: 2.766698, MAE: 1.0543274\n",
      "Epoch 290: MSE: 2.8422406, MAE: 1.0660706\n",
      "Epoch 291: MSE: 2.5701852, MAE: 1.0665181\n",
      "Epoch 292: MSE: 2.6564603, MAE: 1.0439748\n",
      "Epoch 293: MSE: 2.7188697, MAE: 1.035502\n",
      "Epoch 294: MSE: 2.767137, MAE: 1.0577658\n",
      "Epoch 295: MSE: 2.4964128, MAE: 1.0419745\n",
      "Epoch 296: MSE: 2.757732, MAE: 1.0986671\n",
      "Epoch 297: MSE: 2.5399196, MAE: 1.114225\n",
      "Epoch 298: MSE: 2.995366, MAE: 1.0679705\n",
      "Epoch 299: MSE: 2.3586376, MAE: 1.0071737\n",
      "Epoch 300: MSE: 2.6479769, MAE: 1.1450927\n",
      "Epoch 301: MSE: 2.6023037, MAE: 1.0298196\n",
      "Epoch 302: MSE: 2.2329862, MAE: 0.99655116\n",
      "Epoch 303: MSE: 2.843422, MAE: 1.1370443\n",
      "Epoch 304: MSE: 2.4899, MAE: 0.9899544\n",
      "Epoch 305: MSE: 2.67981, MAE: 1.0378929\n",
      "Epoch 306: MSE: 2.6052203, MAE: 1.0919483\n",
      "Epoch 307: MSE: 2.4218466, MAE: 1.0848315\n",
      "Epoch 308: MSE: 2.900044, MAE: 1.0857263\n",
      "Epoch 309: MSE: 2.1795774, MAE: 0.9326693\n",
      "Epoch 310: MSE: 2.8510103, MAE: 1.0971795\n",
      "Epoch 311: MSE: 2.3558362, MAE: 1.014168\n",
      "Epoch 312: MSE: 2.816517, MAE: 1.0792472\n",
      "Epoch 313: MSE: 2.4840987, MAE: 0.9669653\n",
      "Epoch 314: MSE: 2.680021, MAE: 1.0418317\n",
      "Epoch 315: MSE: 2.3006063, MAE: 0.96270525\n",
      "Epoch 316: MSE: 2.3888872, MAE: 1.0553638\n",
      "Epoch 317: MSE: 2.7177556, MAE: 1.0542645\n",
      "Epoch 318: MSE: 2.2048337, MAE: 0.9423412\n",
      "Epoch 319: MSE: 2.7037718, MAE: 1.1164831\n",
      "Epoch 320: MSE: 2.5306127, MAE: 1.0305136\n",
      "Epoch 321: MSE: 2.47349, MAE: 1.0369544\n",
      "Epoch 322: MSE: 2.0784957, MAE: 0.93396413\n",
      "Epoch 323: MSE: 2.8784297, MAE: 1.1576746\n",
      "Epoch 324: MSE: 2.1895368, MAE: 1.0022298\n",
      "Epoch 325: MSE: 2.6578407, MAE: 1.1179578\n",
      "Epoch 326: MSE: 2.254325, MAE: 1.0132198\n",
      "Epoch 327: MSE: 2.6856067, MAE: 1.0263813\n",
      "Epoch 328: MSE: 2.5206912, MAE: 1.0251639\n",
      "Epoch 329: MSE: 2.3049874, MAE: 1.0427438\n",
      "Epoch 330: MSE: 2.5700755, MAE: 1.0500587\n",
      "Epoch 331: MSE: 2.5951974, MAE: 0.9902335\n",
      "Epoch 332: MSE: 2.0135577, MAE: 0.94239813\n",
      "Epoch 333: MSE: 2.5983338, MAE: 1.0753487\n",
      "Epoch 334: MSE: 2.3652139, MAE: 1.0073285\n",
      "Epoch 335: MSE: 2.3540742, MAE: 1.0134287\n",
      "Epoch 336: MSE: 2.1783571, MAE: 1.0290195\n",
      "Epoch 337: MSE: 2.259284, MAE: 1.0515609\n",
      "Epoch 338: MSE: 2.3439522, MAE: 0.98888373\n",
      "Epoch 339: MSE: 2.239645, MAE: 1.0293368\n",
      "Epoch 340: MSE: 2.249991, MAE: 1.0592086\n",
      "Epoch 341: MSE: 2.51545, MAE: 1.1088544\n",
      "Epoch 342: MSE: 2.1201937, MAE: 0.8938169\n",
      "Epoch 343: MSE: 2.5846987, MAE: 1.125812\n",
      "Epoch 344: MSE: 2.1786308, MAE: 1.0125178\n",
      "Epoch 345: MSE: 2.3294573, MAE: 0.94214207\n",
      "Epoch 346: MSE: 2.4841783, MAE: 1.0588152\n",
      "Epoch 347: MSE: 2.1025572, MAE: 0.8794333\n",
      "Epoch 348: MSE: 2.601926, MAE: 1.1206019\n",
      "Epoch 349: MSE: 2.132429, MAE: 0.9706205\n",
      "Epoch 350: MSE: 2.3963668, MAE: 1.0632465\n",
      "Epoch 351: MSE: 2.0145588, MAE: 0.9358846\n",
      "Epoch 352: MSE: 2.1890829, MAE: 1.0571284\n",
      "Epoch 353: MSE: 2.3262289, MAE: 0.93378186\n",
      "Epoch 354: MSE: 2.4395816, MAE: 1.0379367\n",
      "Epoch 355: MSE: 2.4634852, MAE: 1.0137902\n",
      "Epoch 356: MSE: 2.0497193, MAE: 0.9085755\n",
      "Epoch 357: MSE: 2.195309, MAE: 1.0490551\n",
      "Epoch 358: MSE: 2.3841066, MAE: 1.0682517\n",
      "Epoch 359: MSE: 2.1863775, MAE: 0.97229344\n",
      "Epoch 360: MSE: 2.2409508, MAE: 0.9790436\n",
      "Epoch 361: MSE: 2.4902349, MAE: 1.0373104\n",
      "Epoch 362: MSE: 2.5548913, MAE: 1.0650666\n",
      "Epoch 363: MSE: 2.2675014, MAE: 0.8798578\n",
      "Epoch 364: MSE: 1.9882913, MAE: 0.89229053\n",
      "Epoch 365: MSE: 2.1959307, MAE: 1.0423658\n",
      "Epoch 366: MSE: 2.1507618, MAE: 1.0353713\n",
      "Epoch 367: MSE: 2.2659407, MAE: 1.0032746\n",
      "Epoch 368: MSE: 2.23489, MAE: 1.0044267\n",
      "Epoch 369: MSE: 2.1486218, MAE: 0.9047502\n",
      "Epoch 370: MSE: 2.26018, MAE: 1.0224624\n",
      "Epoch 371: MSE: 2.1050937, MAE: 0.989275\n",
      "Epoch 372: MSE: 2.4261084, MAE: 1.0396283\n",
      "Epoch 373: MSE: 2.2390132, MAE: 0.9565736\n",
      "Epoch 374: MSE: 2.0937128, MAE: 0.90576065\n",
      "Epoch 375: MSE: 2.2541454, MAE: 1.015543\n",
      "Epoch 376: MSE: 2.581106, MAE: 0.97585297\n",
      "Epoch 377: MSE: 1.7642366, MAE: 0.8664329\n",
      "Epoch 378: MSE: 2.2938242, MAE: 1.0754704\n",
      "Epoch 379: MSE: 1.9677775, MAE: 0.93085057\n",
      "Epoch 380: MSE: 2.5512564, MAE: 1.0231502\n",
      "Epoch 381: MSE: 2.197184, MAE: 0.9175237\n",
      "Epoch 382: MSE: 2.009687, MAE: 0.898875\n",
      "Epoch 383: MSE: 2.273479, MAE: 1.052901\n",
      "Epoch 384: MSE: 1.9594058, MAE: 0.93083245\n",
      "Epoch 385: MSE: 2.0430722, MAE: 0.84490895\n",
      "Epoch 386: MSE: 2.3420668, MAE: 1.0315745\n",
      "Epoch 387: MSE: 2.0591512, MAE: 0.9257001\n",
      "Epoch 388: MSE: 1.8511548, MAE: 0.88363147\n",
      "Epoch 389: MSE: 2.420221, MAE: 1.004332\n",
      "Epoch 390: MSE: 1.9875458, MAE: 0.9969437\n",
      "Epoch 391: MSE: 2.4225893, MAE: 0.9912536\n",
      "Epoch 392: MSE: 1.7699645, MAE: 0.86357117\n",
      "Epoch 393: MSE: 2.228837, MAE: 0.9890841\n",
      "Epoch 394: MSE: 1.9693921, MAE: 0.93666434\n",
      "Epoch 395: MSE: 2.2272768, MAE: 0.91015565\n",
      "Epoch 396: MSE: 2.0044234, MAE: 0.93574405\n",
      "Epoch 397: MSE: 2.1442773, MAE: 1.0288489\n",
      "Epoch 398: MSE: 1.9601748, MAE: 0.945506\n",
      "Epoch 399: MSE: 2.0004973, MAE: 0.95585907\n",
      "Epoch 400: MSE: 1.9275473, MAE: 1.0469363\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 401: MSE: 2.1391423, MAE: 0.9834826\n",
      "Epoch 402: MSE: 1.9963524, MAE: 0.93260866\n",
      "Epoch 403: MSE: 1.7422082, MAE: 0.87208134\n",
      "Epoch 404: MSE: 2.0251353, MAE: 1.0099846\n",
      "Epoch 405: MSE: 2.4713483, MAE: 0.97791034\n",
      "Epoch 406: MSE: 1.8526359, MAE: 0.8903376\n",
      "Epoch 407: MSE: 1.6257775, MAE: 0.7960783\n",
      "Epoch 408: MSE: 1.8892199, MAE: 1.1056974\n",
      "Epoch 409: MSE: 2.142658, MAE: 0.99807674\n",
      "Epoch 410: MSE: 2.1261678, MAE: 0.9529511\n",
      "Epoch 411: MSE: 2.2038379, MAE: 1.0133382\n",
      "Epoch 412: MSE: 2.1065068, MAE: 0.94087833\n",
      "Epoch 413: MSE: 1.9456577, MAE: 0.8830353\n",
      "Epoch 414: MSE: 1.987342, MAE: 0.94544154\n",
      "Epoch 415: MSE: 2.1662872, MAE: 0.9680983\n",
      "Epoch 416: MSE: 2.1216204, MAE: 0.8896621\n",
      "Epoch 417: MSE: 2.1535184, MAE: 0.9703977\n",
      "Epoch 418: MSE: 1.6284555, MAE: 0.80591166\n",
      "Epoch 419: MSE: 2.6078799, MAE: 1.0939535\n",
      "Epoch 420: MSE: 1.885802, MAE: 0.8518865\n",
      "Epoch 421: MSE: 1.8285732, MAE: 0.87248856\n",
      "Epoch 422: MSE: 2.1396022, MAE: 0.97532934\n",
      "Epoch 423: MSE: 1.8054941, MAE: 0.9314222\n",
      "Epoch 424: MSE: 1.898833, MAE: 0.9378834\n",
      "Epoch 425: MSE: 2.0587332, MAE: 1.0972165\n",
      "Epoch 426: MSE: 2.3773735, MAE: 0.95672965\n",
      "Epoch 427: MSE: 1.8734396, MAE: 0.8821529\n",
      "Epoch 428: MSE: 1.7187449, MAE: 0.90104127\n",
      "Epoch 429: MSE: 2.0920625, MAE: 0.98305094\n",
      "Epoch 430: MSE: 2.0481634, MAE: 0.9285543\n",
      "Epoch 431: MSE: 1.5674127, MAE: 0.91407716\n",
      "Epoch 432: MSE: 2.3057082, MAE: 0.9368512\n",
      "Epoch 433: MSE: 1.6059024, MAE: 0.8918092\n",
      "Epoch 434: MSE: 2.1378045, MAE: 1.0082663\n",
      "Epoch 435: MSE: 1.8069695, MAE: 0.9383101\n",
      "Epoch 436: MSE: 1.932408, MAE: 0.92350316\n",
      "Epoch 437: MSE: 1.7996694, MAE: 0.9497357\n",
      "Epoch 438: MSE: 1.7650225, MAE: 0.8501568\n",
      "Epoch 439: MSE: 2.2569108, MAE: 1.0178587\n",
      "Epoch 440: MSE: 2.0368824, MAE: 0.93046963\n",
      "Epoch 441: MSE: 1.6586083, MAE: 0.8774116\n",
      "Epoch 442: MSE: 1.6638896, MAE: 0.8656534\n",
      "Epoch 443: MSE: 2.0320647, MAE: 0.96143985\n",
      "Epoch 444: MSE: 2.2150764, MAE: 0.9589602\n",
      "Epoch 445: MSE: 2.1071959, MAE: 0.907773\n",
      "Epoch 446: MSE: 1.7532563, MAE: 0.88580954\n",
      "Epoch 447: MSE: 2.1588764, MAE: 0.9271664\n",
      "Epoch 448: MSE: 1.9729767, MAE: 0.9257062\n",
      "Epoch 449: MSE: 1.6189989, MAE: 0.7541823\n",
      "Epoch 450: MSE: 1.9455845, MAE: 0.95737547\n",
      "Epoch 451: MSE: 2.1335454, MAE: 0.91260326\n",
      "Epoch 452: MSE: 1.6346242, MAE: 0.8267285\n",
      "Epoch 453: MSE: 1.8054204, MAE: 0.94459987\n",
      "Epoch 454: MSE: 2.0148292, MAE: 1.0225977\n",
      "Epoch 455: MSE: 2.022909, MAE: 0.8914356\n",
      "Epoch 456: MSE: 1.8515967, MAE: 0.9112024\n",
      "Epoch 457: MSE: 1.6702584, MAE: 0.87819964\n",
      "Epoch 458: MSE: 2.2747612, MAE: 0.9970331\n",
      "Epoch 459: MSE: 1.6699033, MAE: 0.80945176\n",
      "Epoch 460: MSE: 1.8963745, MAE: 0.98044056\n",
      "Epoch 461: MSE: 2.016756, MAE: 0.95894885\n",
      "Epoch 462: MSE: 1.7279505, MAE: 0.87328655\n",
      "Epoch 463: MSE: 2.0527582, MAE: 0.9154061\n",
      "Epoch 464: MSE: 1.5434761, MAE: 0.81517893\n",
      "Epoch 465: MSE: 1.9222476, MAE: 0.9552828\n",
      "Epoch 466: MSE: 1.8396833, MAE: 0.9567124\n",
      "Epoch 467: MSE: 1.5861919, MAE: 0.8737236\n",
      "Epoch 468: MSE: 2.062336, MAE: 0.9520855\n",
      "Epoch 469: MSE: 1.8852284, MAE: 0.848645\n",
      "Epoch 470: MSE: 1.9771957, MAE: 0.93317854\n",
      "Epoch 471: MSE: 1.7079604, MAE: 0.9183467\n",
      "Epoch 472: MSE: 1.884468, MAE: 0.7979444\n",
      "Epoch 473: MSE: 1.963796, MAE: 1.0059327\n",
      "Epoch 474: MSE: 1.3903452, MAE: 0.7476398\n",
      "Epoch 475: MSE: 2.0456457, MAE: 1.0286996\n",
      "Epoch 476: MSE: 1.5425693, MAE: 0.90627736\n",
      "Epoch 477: MSE: 2.0112476, MAE: 0.98892814\n",
      "Epoch 478: MSE: 1.4354928, MAE: 0.7733854\n",
      "Epoch 479: MSE: 2.093851, MAE: 1.0760102\n",
      "Epoch 480: MSE: 1.954952, MAE: 0.8517307\n",
      "Epoch 481: MSE: 1.5023459, MAE: 0.81447184\n",
      "Epoch 482: MSE: 1.955289, MAE: 0.8791892\n",
      "Epoch 483: MSE: 1.80357, MAE: 0.930727\n",
      "Epoch 484: MSE: 1.808779, MAE: 0.9637473\n",
      "Epoch 485: MSE: 1.769795, MAE: 0.88276947\n",
      "Epoch 486: MSE: 1.7752646, MAE: 0.9061363\n",
      "Epoch 487: MSE: 1.5515298, MAE: 0.81836665\n",
      "Epoch 488: MSE: 1.6913869, MAE: 0.90961325\n",
      "Epoch 489: MSE: 1.8748481, MAE: 1.067188\n",
      "Epoch 490: MSE: 2.0368338, MAE: 0.9614334\n",
      "Epoch 491: MSE: 1.9930379, MAE: 0.8933137\n",
      "Epoch 492: MSE: 1.6152343, MAE: 0.79594606\n",
      "Epoch 493: MSE: 1.6397833, MAE: 0.93405026\n",
      "Epoch 494: MSE: 1.8326427, MAE: 0.94108045\n",
      "Epoch 495: MSE: 1.6242542, MAE: 0.8328901\n",
      "Epoch 496: MSE: 1.9047801, MAE: 0.9581281\n",
      "Epoch 497: MSE: 1.6817826, MAE: 0.84436214\n",
      "Epoch 498: MSE: 1.6671867, MAE: 0.8551764\n",
      "Epoch 499: MSE: 1.7665759, MAE: 1.0199685\n",
      "Epoch 500: MSE: 1.6138743, MAE: 0.8525842\n"
     ]
    }
   ],
   "source": [
    "for epoch in 1...epochCount {\n",
    "    var epochLoss: Float = 0\n",
    "    var epochMAE: Float = 0\n",
    "    var batchCount: Int = 0\n",
    "    var batchArray = Array(repeating: false, count: numberOfBatch)\n",
    "    for batch in 0..<numberOfBatch {\n",
    "        var r = batch\n",
    "        if shuffle {\n",
    "            while true {\n",
    "                r = Int.random(in: 0..<numberOfBatch)\n",
    "                if !batchArray[r] {\n",
    "                    batchArray[r] = true\n",
    "                    break\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "        \n",
    "        let batchStart = r * batchSize\n",
    "        let batchEnd = min(numTrainRecords, batchStart + batchSize)\n",
    "        let (loss, grad) = model.valueWithGradient { (model: RegressionModel) -> Tensor<Float> in\n",
    "            let multiInput = MultiInputs(numerical: [XNumericalTrain[batchStart..<batchEnd]],\n",
    "                                         categorical: [XCategoricalTrain[0][batchStart..<batchEnd],\n",
    "                                                       XCategoricalTrain[1][batchStart..<batchEnd]])\n",
    "            let logits = model(multiInput)\n",
    "            return meanSquaredError(predicted: logits, expected: YTrain[batchStart..<batchEnd])\n",
    "        }\n",
    "        optimizer.update(&model, along: grad)\n",
    "        \n",
    "        let multiInput = MultiInputs(numerical: [XNumericalTrain[batchStart..<batchEnd]],\n",
    "                                     categorical: [XCategoricalTrain[0][batchStart..<batchEnd],\n",
    "                                                   XCategoricalTrain[1][batchStart..<batchEnd]])\n",
    "        let logits = model(multiInput)\n",
    "        epochMAE += mae(predictions: logits, truths: YTrain[batchStart..<batchEnd])\n",
    "        epochLoss += loss.scalarized()\n",
    "        batchCount += 1\n",
    "    }\n",
    "    epochMAE /= Float(batchCount)\n",
    "    epochLoss /= Float(batchCount)\n",
    "\n",
    "    print(\"Epoch \\(epoch): MSE: \\(epochLoss), MAE: \\(epochMAE)\")\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 0.09474914, MAE: 0.023811752\r\n"
     ]
    }
   ],
   "source": [
    "Context.local.learningPhase = .inference\n",
    "\n",
    "let multiInputTest = MultiInputs(numerical: [XNumericalTest],\n",
    "                                 categorical: [XCategoricalTest[0],\n",
    "                                               XCategoricalTest[1]])\n",
    "\n",
    "let prediction = model(multiInputTest)\n",
    "\n",
    "let predictionMse = meanSquaredError(predicted: prediction, expected: YTest).scalarized()/Float(numTestRecords)\n",
    "let predictionMae = mae(predictions: prediction, truths: YTest)/Float(numTestRecords)\n",
    "\n",
    "print(\"MSE: \\(predictionMse), MAE: \\(predictionMae)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "S4TF_House.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Swift",
   "language": "swift",
   "name": "swift"
  },
  "language_info": {
   "file_extension": ".swift",
   "mimetype": "text/x-swift",
   "name": "swift",
   "version": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
